Skip to content

feat(fs): enhance S3 filesystem reliability and compatibility - #454

Draft
zjw1111 wants to merge 2 commits into
alibaba:mainfrom
zjw1111:codex/fix-s3-followup
Draft

feat(fs): enhance S3 filesystem reliability and compatibility#454
zjw1111 wants to merge 2 commits into
alibaba:mainfrom
zjw1111:codex/fix-s3-followup

Conversation

@zjw1111

@zjw1111 zjw1111 commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Purpose

Linked issue: #357, follow-up to #433

Address the remaining correctness, robustness, diagnostics, and build issues in the
lightweight S3 filesystem implementation:

  • keep the asynchronous executor alive independently from individual S3 clients
  • preserve virtual-hosted addressing for compatible HTTPS endpoints and correctly handle IPv6
    endpoint authorities
  • parse ListObjectsV2 ISO 8601 modification times
  • add configurable request and low-speed timeouts
  • retain bounded S3 error responses and bucket-region headers for actionable diagnostics
  • cover not-found responses, asynchronous client release, endpoint addressing, timestamps,
    retries, and timeouts
  • declare the required CURL, OpenSSL, and Arrow link dependencies for supported static and
    shared builds

Tests

  • pre-commit run --files CMakeLists.txt cmake_modules/ThirdpartyToolchain.cmake src/paimon/CMakeLists.txt src/paimon/common/fs/http_client.cpp src/paimon/common/fs/http_client.h src/paimon/common/fs/http_client_test.cpp src/paimon/fs/s3/CMakeLists.txt src/paimon/fs/s3/s3_file_system.cpp src/paimon/fs/s3/s3_file_system.h src/paimon/fs/s3/s3_file_system_test.cpp
  • cmake --build build --target paimon-http-client-test paimon-s3-file-system-test -j64
  • ./build/debug/paimon-http-client-test
  • ./build/debug/paimon-s3-file-system-test

API and Format

No public API, storage format, or protocol changes.

Documentation

No.

Generative AI tooling

Generated-by: Codex (GPT-5)

@zjw1111 zjw1111 changed the title fix(fs): address S3 filesystem follow-up issues feat(fs): enhance S3 filesystem reliability and compatibility Jul 28, 2026
@zjw1111

zjw1111 commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator Author

mark draft now, will refactor after #441 merged

@zjw1111
zjw1111 marked this pull request as draft July 30, 2026 03:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant